Single Image Sensor for Capturing Mixed Structured-light Images and Regular Images

ABSTRACT

An integrated image sensor for capturing a mixed structured-light image and regular image using an integrated image sensor are disclosed. The integrated image sensor comprises a pixel array, one or more output circuits, one or more analog-to-digital converters, and one or more timing and control circuits. The timing and control circuits are arranged to perform a set of actions including capturing a regular image and a structured-light image. According to the present invention, the structured-light image captured before or after the regular image is used to derive depth or shape information for the regular image. An endoscope based on the above integrated image sensor is also disclosed. The endoscope may comprises a capsule housing adapted to be swallowed, where the components of integrated image sensor, a structured light source and anon-structured light source are enclosed and sealed in the capsule housing.

CROSS REFERENCES

The present application is a divisional application of and claims thepriority to U.S. Non-provisional application Ser. No. 15/871,991, filedon Jan. 16, 2018, which is a divisional application of and claims thepriority to U.S. Non-provisional application Ser. No. 14/884,788, filedon Oct. 16, 2015, now patented as U.S. Pat. No. 9,936,151 on Apr. 3,2018. The U.S. Non-Provisional Applications are incorporated byreference in their entireties.

FIELD OF THE INVENTION

The present invention relates to a single image sensor capable ofcapturing structured-light images and regular image, where thestructured-light image is used to derive depth or shape informationrelated to the corresponding regular image.

BACKGROUND AND RELATED ART

Devices for imaging body cavities or passages in vivo are known in theart and include endoscopes and autonomous encapsulated cameras.Endoscopes are flexible or rigid tubes that pass into the body throughan orifice or surgical opening, typically into the esophagus via themouth or into the colon via the rectum. An image is formed at the distalend using a lens and transmitted to the proximal end, outside the body,either by a lens-relay system or by a coherent fiber-optic bundle. Aconceptually similar instrument might record an image electronically atthe distal end, for example using a CCD or CMOS array, and transfer theimage data as an electrical signal to the proximal end through a cable.Endoscopes allow a physician control over the field of view and arewell-accepted diagnostic tools.

Capsule endoscope is an alternative in vivo endoscope developed inrecent years. For capsule endoscope, a camera is housed in a swallowablecapsule, along with a radio transmitter for transmitting data, primarilycomprising images recorded by the digital camera, to a base-stationreceiver or transceiver and data recorder outside the body. The capsulemay also include a radio receiver for receiving instructions or otherdata from a base-station transmitter. Instead of radio-frequencytransmission, lower-frequency electromagnetic signals may be used. Powermay be supplied inductively from an external inductor to an internalinductor within the capsule or from a battery within the capsule.

An autonomous capsule camera system with on-board data storage wasdisclosed in the U.S. Pat. No. 7,983,458, entitled “In Vivo AutonomousCamera with On-Board Data Storage or Digital Wireless Transmission inRegulatory Approved Band,” granted on Jul. 19, 2011. The capsule camerawith on-board storage archives the captured images in on-boardnon-volatile memory. The capsule camera is retrieved upon its exitingfrom the human body. The images stored in the non-volatile memory of theretrieved capsule camera are then accessed through an output port on inthe capsule camera.

While the two-dimensional images captured by the endoscopes have beenshown useful for diagnosis, it is desirable to be able to capturegastrointestinal (GI) tract images with depth information (i.e.,three-dimensional (3D) images) to improve the accuracy of diagnosis orto ease the diagnosis process. In the field of 3D imaging, 3D images maybe captured using a regular camera for the texture information in thescene and a separate depth camera (e.g. Time of Flight camera) for thedepth information of the scene in the field of view. The 3D images mayalso be captured using multiple cameras, where multiple cameras areoften used in a planar configuration to capture a scene from differentview angles. Then, point correspondence is established among multipleviews for 3D triangulation. Nevertheless, these multi-camera systems maynot be easily adapted to the GI tract environment, where the space isvery limited. In the past twenty years, a structured light technologyhas been developed to derive the depth or shape of objects in the sceneusing a single camera. In the structured light system, a light source,often a projector is used to project known geometric pattern(s) ontoobjects in the scene. A regular camera can be used to capture imageswith and without the projected patterns. The images captured with thestructured light can be used to derive the shapes associated with theobjects in the scene. The depth or shape information is then used withregular images, which are captured with non-structured floodlit light,to create 3D textured model of the objects. The structured lighttechnology has been well known in the field. For example, in“Structured-light 3D surface imaging: a tutorial” (Geng, in Advances inOptics and Photonics, Vol. 3, Issue 2, pp. 128-160, Mar. 31, 2011),structured light technology using various structured light patterns aredescribed and the corresponding performances are compared. In anotherexample, various design, calibration and implement issues are describedin “3-D Computer Vision Using Structured Light: Design, Calibration andImplementation Issues” (DePiero et al., Advances in Computers, Volume43, Jan. 1, 1996, pages 243-278). Accordingly, the details of thestructured light technology are not repeated here.

While the structured light technology may be more suitable for 3Dimaging of the GI tract than other technologies, there are still issueswith the intended application for GI tract. For example, most of thestructured light applications are intended for stationary object.Therefore, there is no object movement between the capturedstructured-light image and the regular image. Nevertheless, in thecapsule camera application for GI tract imaging, both the capsule cameraand the GI parts (e.g. small intestines and colons) may be moving.Therefore, there will be relative movement between the structured-lightimage and the regular image if they are captured consecutively.Furthermore, the capsule camera application is a very power-sensitiveenvironment. The use of structured light will consume system power inaddition to capturing the regular images. Besides, if one image withstructured light is taken after each regular image, the useful framerate will be dropped to half. If the same frame rate of regular imagesis maintained, the system would have to capture images at twice theregular frame rate and consume twice the power in image capture.Accordingly, it is desirable to develop technology for structured lightapplication in the GI tract that can overcome these issues mentionedhere.

BRIEF SUMMARY OF THE INVENTION

An integrated image sensor for capturing a mixed structured-light imageand regular image using an integrated image sensor are disclosed. Theintegrated image sensor comprises a pixel array being responsive tolight energy received by the pixel array to produce pixel signals havinga voltage level depending on the light energy received by the pixelarray; one or more output circuits coupled to the pixel array to accessthe pixel signals produced by the pixel array; one or moreanalog-to-digital converters having a first dynamic range and a seconddynamic range; and one or more timing and control circuits coupled tothe pixel array, said one or more output circuits, said one or moreanalog-to-digital converters or a combination thereof. The timing andcontrol circuits are arranged to capture, by the pixel array, astructured-light image formed on a common image plane during a firstframe period by applying first reset signals to the pixel array to resetrows of pixels of the pixel array, exposing the rows of pixels of thepixel array to first illumination from a structured light source tocause first analog signals from the rows of pixels and converting thefirst analog signals from the rows of pixels of the pixel array intofirst digital outputs for the structured-light image using one or moreanalog-to-digital converters; capture, by the pixel array, a firstregular image formed on the common image plane during a second frameperiod by applying second reset signals to the pixel array to reset therows of pixels of the pixel array, exposing the rows of pixels to secondillumination from a non-structured light source to cause second analogsignals from the rows of pixels, and converting the second analogsignals from the rows of pixels into second digital outputs for thefirst regular image using said one or more analog-to-digital converters;capture, by the pixel array, a second regular image formed on the commonimage plane during a third frame period by applying third reset signalsto the pixel array to reset the rows of pixels of the pixel array,exposing the rows of pixels to the second illumination from thenon-structured light source to cause third analog signals from the rowsof pixels, and converting the third analog signals from the rows ofpixels into third digital outputs for the second regular image usingsaid one or more analog-to-digital converters; and combine the seconddigital outputs and the third digital outputs to form a combined regularimage. The structured-light image is captured between the first regularimage and the second regular image to derive depth or shape informationfor the combined regular image.

An endoscope based on the above integrated image sensor is alsodisclosed. The endoscope further comprises a structured light source anda non-structured light source, which may be coupled to the timing andcontrol circuits.

A camera system based on the above integrated image sensor is alsodisclosed. The camera further comprises a structured light source and anon-structured light source, which may be coupled to the timing andcontrol circuits.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates exemplary analog-to-digital converter with 8-bitdynamic range.

FIGS. 2A-B illustrate ramp voltage waveforms used as a reference signalfor the analog-to-digital converter, where the waveforms shown are for9-bit (FIG. 2A) and 6-bit (FIG. 2B) dynamic resolutions.

FIG. 3 illustrates an exemplary block diagram of a pixel array forcapturing a tandem structured-light image and regular image according toan embodiment of the present invention.

FIG. 4A illustrates an exemplary timing chart for capturing two regularimages.

FIG. 4B illustrates an example of a tandem structured-light image andregular image according to an embodiment of the present invention.

FIG. 5 illustrates an exemplary timing corresponding to FIG. 4B forapplying the structured light for capturing a structured-light imageaccording to an embodiment of the present invention, where thestructured light duration is explicitly shown and some notations aredeleted for clear illustration.

FIG. 6 illustrates another view of applying the structured light forcapturing a structured-light image and capturing a regular imageaccording to an embodiment of the present invention.

FIG. 7A illustrates exemplary timing charts for capturing a two-sessionimage according to an embodiment of the present invention, where a firstregular sub-image is captured in the first session and a mixed imageconsisting of a structured-light image and a second regular sub-image iscaptured in the second session, the regular image are combined as thefinal output and the integration period for the first session isapproximately the same as the integration period for the second session.

FIG. 7B illustrates exemplary timing charts similar to these of FIG. 7A,where the integration period for the first session is approximatelythree times as long as the integration period for the second session.

FIG. 8 illustrates an exemplary flowchart for capturing a tandemstructured-light image and regular image according to an embodiment ofthe present invention, where the structured-light image has lowerdynamic range than the regular image.

FIG. 9 illustrates an exemplary flowchart for capturing a tandemstructured-light image and regular image according to another embodimentof the present invention, where the structured-light image has lowerdynamic range than the regular image.

DETAILED DESCRIPTION OF THE INVENTION

It will be readily understood that the components of the presentinvention, as generally described and illustrated in the figures herein,may be arranged and designed in a wide variety of differentconfigurations. Thus, the following more detailed description of theembodiments of the systems and methods of the present invention, asrepresented in the figures, is not intended to limit the scope of theinvention, as claimed, but is merely representative of selectedembodiments of the invention. References throughout this specificationto “one embodiment,” “an embodiment,” or similar language mean that aparticular feature, structure, or characteristic described in connectionwith the embodiment may be included in at least one embodiment of thepresent invention. Thus, appearances of the phrases “in one embodiment”or “in an embodiment” in various places throughout this specificationare not necessarily all referring to the same embodiment.

Furthermore, the described features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments. Oneskilled in the relevant art will recognize, however, that the inventioncan be practiced without one or more of the specific details, or withother methods, components, etc. In other instances, well-knownstructures, or operations are not shown or described in detail to avoidobscuring aspects of the invention. The illustrated embodiments of theinvention will be best understood by reference to the drawings, whereinlike parts are designated by like numerals throughout. The followingdescription is intended only by way of example, and simply illustratescertain selected embodiments of apparatus and methods that areconsistent with the invention as claimed herein.

Endoscopes are normally inserted into the human body through a naturalopening such as the mouth or anus. Therefore, endoscopes are preferredto be small sizes so as to be minimally invasive. To derive or capturethe depth or shape information while capturing live images or videos ofthe GI tract with endoscopes, it is crucial to maintain the small-sizeform factor. Besides, with the small size and the capability to capturedepth information along with corresponding images or video, such cameraalso finds its applications in other applications requiring compactsize, such as a wearable devices.

One technique that may capture depth information is to use a colorfilter placed on top of selected sensor pixels with the passbandreasonably narrow and capture the color information and depthinformation simultaneously. The environment light sources with spectrumin the filter passband will cause negligible amount of energy projectedonto the sensor. For the case of RGB pixels, a fourth type of pixels maybe added to capture light with the spectrum in the passband of thefilter placed on top of these pixels. Then, the structured light thathas the spectrum substantially in the passband can be projected onto thescene. However this approach will reduce the spatial resolution of theimages or video captured using such image sensor.

Another technique is to obtain the depth information as well as 3Dtopology by projecting structured light patterns that are visible in theRGB sensors. However the real time image and/or video will be confoundedby the structured light superimposed on it. This invention describesmethods to use a single camera to achieve depth information by using thestructured light approach while taking images or real time video usingthe camera.

As mentioned before, a conventional structured light approach with asingle camera would incur several drawbacks. For example, the camerawith a frame rate of 30 frames per second may be used. A conventionalapproach would take live video with interleaved images corresponding toimages with and without the structured light. One issue is that thedepth information is 1/30 second away from corresponding images to beviewed. If there is any movement in the scene, the depth information maynot accurately represent the 3D topology of the corresponding images at1/30 second away. In addition, the effective frame rate for the video tobe viewed is dropped to 15 frames per second in this example.

In some video applications, the frame rate is crucial for the intendedapplication. For example, a high frame-rate camera with frame rate inthe 100's per second or more is required to capture video of fast movingobjects such as a travelling bullet. In this case, the use of structuredlight would cut the frame rate to half and may hinder the intendedapplication. For a capsule camera, the video for the gastrointestinal(GI) tract is normally a few frames per second and the camera could beoperating at twice the original frame rate to compensate the reductionof effective frame rate due to capturing structured-light images.However, it would result in twice as much power consumption, which isnot desirable in the power-limited capsule environment.

Each frame rate has a corresponding frame period. During the frameperiod, the sensor will spend a subset of the frame period foraccumulating charges emitted in response to incidental light on thesensor. The integration time must be sufficiently small so that theimage is substantially stationary to avoid causing any motion blur inthe captured image.

FIG. 1 illustrates an example of ADC (analog-to-digital converter) thatcan be used for an image sensor. The ADC includes a comparator (110) tocompare an input analog signal (120) with a reference signal (130). Theanalog signal (120) from the analog circuits connected to a pixel iscoupled to one terminal of the comparator to compare with a rampvoltage. In one embodiment, the ramp voltage can be generated using aclock that is used for digital logic circuits so that the ramp voltagegoes one step higher according to each driving clock. FIG. 2A and FIG.2B show two examples of ramp voltage corresponding to 9-bit (i.e., 512levels) and 6-bit (i.e., 64 levels) respectively. The same clocksupplied to generate the ramp voltage is also counted by the counter(140). When the ramp voltage level reaches the analog signal (120) fromthe pixel analog readout circuits connected to a pixel, the comparatoroutput will switch (e.g. from high to low or low to high) to indicatethe event. At the same time, the comparator output signal will triggerthe registers (150) so as to latch a counter value that counts thenumber of clocks indicating the ramp voltage value crossing the analogsignal (120). The output from the pixel is often measured twice usingthe correlated double sampling (CDS) technique, which is well known inthe field to take care of the fixed pattern noise due to manufacturingvariation among pixels. The offset reset signal can be removed usinganalog circuits (e. g. correlated double sampling) or digital circuits.For digital implementation, the digital counter readout after reset canbe subtracted from the digital counter readout of a pixel after anintegration period.

There are several factors determining how fast the pixel can accumulateelectronic charges and how fast the signal can be readout. As shown inthe example of FIG. 1, the analog output signal (120) from analogcircuits connected to each pixel is compared to a reference signal(i.e., the voltage ramp). Depending on the desired digital pixelresolution (e.g. 9 bits vs 6 bits), a corresponding ramp signal can begenerated. The readout speed will depend on the speed of the counter,the comparator and other related circuits. The ramp signal correspondingto higher digital resolution (i.e., higher dynamic range) will takelonger time to generate. The implementation to support 6-bit digitaldynamic range as shown in FIG. 2B will be 8 times faster than theimplementation for digital 9-bit dynamic range as shown in FIG. 2A.

There are other variations to implement ADC, such as successiveapproximation ADC. For the successive approximation ADC, the referencevoltage starts with a coarse level. Depending on whether the inputvoltage is higher or lower than the reference voltage, the referencevoltage is refined by increasing or decreasing the previous referencevoltage by half of a previous voltage interval. The refined referencevoltage is used as a current reference voltage for next successivecomparison. The process is terminated until a desired resolution isachieved. In each round of successive approximation, one bit is used toindicate whether the input voltage is higher or lower than the referencevoltage. Accordingly, the ADC resolution corresponds to the number ofsuccessive approximation of the successive-approximation ADC. Ingeneral, the higher the dynamic ranges, the longer the readout will be.Not only more comparisons will be required, but also the voltage willtake longer time to settle down since the accuracy requirements of theramp up voltage or reference voltages are high. The sensor array has alarge intrinsic RC constant, which takes time to settle to within thelimits required by the accuracy. In the case of a high dynamic range,the conductor line carrying the reference voltage (i.e., the rampreference signal) requires more time to settle due to the inductance,along with R (resistance) and C (capacitance), of the conductor line.The length of the conductor line for the sensor array usually is in theorder of 1,000's μm (micro meter), which may result in an inductancearound a few nH (nano Henry). Unlike resistance, the inductance will notbe scaled down inversely proportional to the conductor cross section.The high dynamical range is an important factor for image/video qualityto provide detailed shades of the objects in the scene. On the otherhand, the images for structured light pattern are mainly used to derivedepth/shape information based on the geometric information of knowpatterns, such as grids. The important information to be derived isrelated to the locations of the grid lines. Accordingly, the requirementon the dynamic range is substantially lower than that for the regularimages to be viewed by the human eyes.

Since the required dynamic range for the structured-light image is muchless than that for a regular image, the present invention takesadvantage of the different dynamic range requirements to shorten theframe period for the structured-light image. FIG. 3 illustrates asimplified system block diagram of an integrated image sensorincorporating an embodiment of the present invention. The integratedimage sensor comprises a pixel array (310) being responsive to lightenergy received by the pixel array to produce signal data having avoltage level depending on the light energy received, output circuits(320) coupled to the pixel array to access the signal data produced bythe pixel array, one or more analog-to-digital converters (ADCs, 330)having a first dynamic range and a second dynamic range, andtiming/control circuits (340 a and 340 b). The pixel array may consistof monochrome pixels or color pixels. The pixel array can be based onthe CMOS technology or the CCD technology. The output circuits arecoupled to the pixel array under the control of the timing/controlcircuits. For example, the pixel array outputs can be transferred to theoutput circuits row by row under the control of the timing/controlcircuits. The output circuits may also include amplifier and CDScircuit, where the CDS circuit is used to take care of the offset inindividual pixels after reset. While the timing/control circuits (340 aand 340 b) are shown as two separate blocks, they may also beimplemented as a unified block.

The ADC circuit(s) is capable of operating at a first dynamic range anda second dynamic range. The first dynamic range is smaller than thesecond dynamic range. For example, the first dynamic range maycorrespond to 6 bits and the second dynamic range may correspond to 9bits. Individual ADCs with different dynamic ranges may be used. Sincethe structured-light image and the regular image are captured in serialinstead of parallel, a single ADC with configurable dynamic range mayalso be used. For example, an adaptively configurable ADC is disclosedin U.S. Pat. No. 8,369,458 issued to Wong et al. on Feb. 5, 2013. Thetiming/control circuits may include row scan circuit and column scancircuit. The timing/control circuits are also responsible to generatevarious control signals such as reset signals. In the following,preferred embodiments are provided regarding configuring the imagesensor to capture structured-light images and regular images.

FIG. 4A illustrates an example of the timing for a regular image sensor,where the row timing is shown from frame to frame in the horizontaldirection and from the first row (top) to the last row (bottom) in thevertical direction. In a typical image sensor, the image pixels are readout row by row. There are several phases of operation for each rowincluding row reset, integration and readout. The operations usually arestaggered from row to row as shown in FIG. 4A. However, the readoutoperation for a current row has to wait until the previous row readoutis complete. For a regular image sensor, the similar timing patternsjust repeat from frame to frame with the same frame time. The periodtime is the same for all frames. FIG. 4A illustrates the general timingdiagram for a typical image sensor. The timing charts may not be drawnto scale.

FIG. 4B illustrates an example of the timing for capturing a mixedstructured-light image and regular image according to one embodiment ofthe present invention, where the structured light is taken with smallerdynamic range. The frame duration for the structured-light image is muchshorter than the frame duration of the following regular image asillustrated in FIG. 4B. The timing charts in FIG. 4B are not drawn toscale, particularly for the structured light section. The section forthe structured light has been expanded in order to illustrate moredetails. In FIG. 4B, the timing signals associated with thestructured-light image are shown in Bold-Italic font. The less dynamicrange can be achieved, for example, by fewer comparisons with thereference voltage generated by ramping up voltage with a larger step asillustrated in FIGS. 2A and 2B. Alternatively, the reference voltage canbe generated with a smaller ramp range between the starting voltage andthe ending voltage. Since the readout is faster for thisstructured-light image than the regular image, the structured-lightimage could be squeezed within the reset timing period of the regularimage. Since the structure light image does not need a dynamic range ashigh as the regular image, the structure light image reset does not needto be as thorough as that for regular image. Therefore, a shorter resettime could be used for the structured light. Accordingly, in oneembodiment, the minimum row reset time among the rows of pixels of theimage sensor for the structured-light image can be substantially shorterthan the minimum row reset time among the rows of pixels of the imagesensor for the regular image.

FIG. 5 is identical to FIG. 4B with some notations eliminated toconveniently demonstrate the timing setup. The duration for thestructured light is indicated. The structured light duration could besubstantially close to the end of first row integration phase for thestructured light. On the other hand, the structured light duration mustbe within the integration phase of the last row for the structured-lightimage. The optimal structure light pulse duration is from the start oflast row integration to the start of first row readout. In order tospeed up the operation of capturing a structured-light image, as well asto reduce the impact of the energy of the other light sources, such asambient light, in the scene entering the camera, the structured lightduration is set to be very short. If the integration requires a longertime than desired, the structural light pulse energy can be increased byincreasing either the intensity or the duration. In a preferredembodiment, the structured light intensity is substantially strongercompared with the light intensity from the regular light source orambient light. However, the structured light can be applied for asubstantially shorter period than the human visual retention time,making the image of the scene exposed by non-structured light relativelyweaker and the signal/noise ratio much stronger for the structure light.The non-structured light may correspond to a broadband, narrowband, orfluoroscopic light. The energy of the structure light pulse may belimited by system power constraints or by a desire not to annoy,distract, or eye-damage persons in the scene exposed to the structuredlight. Another energy limitation may be a tissue-damage optical-energythreshold in medical-imaging applications. The sensor analog gain may beincreased so that the pixel luminance resulting from structured lightillumination is well above background noise and luminance from otherlight sources while the structured light energy is at an acceptably lowlevel. Increased gain results in greater noise amplification and reducedsensor dynamic range.

While the main intended application is for the GI tract, the usage ofshort-duration and high-intensity structured light can also benefit thenon-GI applications. For example, the present invention may also beapplied to conventional photography to capture mixed regular images andstructured-light images of natural scenes and use the depth or shapeinformation derived from the structured images to render 3D images ofthe scene. In order to derive more reliable depth or shape informationusing the structured-light images, it is desirable to select astructured-light source having light spectrum very different from thecolor spectrum of the underlying scene captured under ambient light orilluminated by one or more non-structured lights.

While there are readout schemes that may start to read the higher (i.e.,most significant) bits during integration, the readout with a largerdynamic range will take a longer time to complete the readout. This isdue to more comparisons and longer reference voltage settling timeneeded. Accordingly, reducing the dynamic range for the structured-lightimage will be able to reduce the row processing duration. This is alsotrue for image sensors operated in the global shutter mode. Accordingly,the settling time associated with a reference voltage, provided to theanalog-to-digital converters to compare with an input analog voltage, isshorter for the first structured-light image than for the regular image.Due to the less accuracy required, the reset signal does not need to beheld for so long to reset the pixels and/or related circuits to anoptimal level.

As shown in FIG. 5, the structured light is preferred to be applied at aspecific time so that the duration is short, but the short durationcovers at least a portion of integration time for all pixel rows. Thecontrol signal to trigger the structured light may be derived from theframe signal and the clock. However, the control signal for thestructured light may also come from other module of the integratedsensor. In another implementation, a fixed or programmable delay for thecontrol signal from the sensor can be used to adjust the timings tomatch optimal system design.

In one embodiment, the structured light is generated using multiplelight sources with at least two different colors or patterns. By usingmultiple colors, a color or a combination of colors can be selected tocause the light spectrum of the selected color(s) substantiallydifferent from the color spectrum of associated with regular images ofan anticipated scene illuminated by the non-structured light or underambient light. The spectrum associated with the structured light can besubstantially distinct from the spectrum associated with regular imagesof an anticipated scene. The image sensor may correspond to a colorimage sensor comprising at least first and second color pixels arrangedin a mosaic pattern, and the spectrum associated with the structuredlight can be substantially concentrated on the spectrum of first orsecond color pixels. The structured-light image can be captured atreduced spatial resolution by reading out only selected digital outputsrelated to the first or second pixels having spectrum substantiallycorresponding to the structured-light spectrum.

For capsule applications, the integrated image sensor may be inside asealed capsule housing for imaging gastrointestinal tract of human body.Since there is no ambient light in the GI tract, the capsule device hasto provide both the structured light and the lighting for regularimages. In this case, the structured light sources for structured lightimages and the illumination light sources for regular images can besealed in the housing.

FIG. 6 illustrates another perspective of the timing chartscorresponding to these in FIG. 5, where the image readouts for thestructured image and regular images are highlighted using differentfill-patterns. The timing signals related to the structured-light imageare indicated in FIG. 6, where sli-reset corresponds to thestructured-light image reset, sli-integration corresponds to thestructured-light image integration and sli-readout corresponds to thestructured-light image readout. The structured light duration is alsoindicated in FIG. 6, where the structured light is applied during thesli-integration. The whole period for capturing the structured-lightimage may occur during the reset period of a regular image reset. In theexample shown in FIG. 6, the reset for the structure light image may bevery brief compared to the reset period for a regular image. The timingcharts in FIG. 6 illustrate the cycles for capturing a tandemstructured-light image and regular image, where the timing for theregular image is modified. In particular, the reset period of theregular image capture is substantially reduced to accommodate thecapture of the structured-light image. Accordingly, the timing chartsincorporating an embodiment of the present invention shown in FIG. 6 canbe considered as two separate images or a combo of structured-lightimage and timing-modified regular image.

In the embodiments disclosed above, a structured light image is capturedtemporally close to a regular image so as to provide more accuratedepth/shape information for the associated regular image. In anotherembodiment, a two-session capture is disclosed, where the regular imageis split into two sub-images with a structured light image captured inbetween. The regular integration time for the regular image is splitbetween the two sub-images. The digital outputs for the two regularsub-images are combined to form a regular image output. This approachhas several advantages. First, each sub-image is converted into digitaloutputs using the ADC that is used for the regular image. Accordingly,each sub-image will have the same dynamic range as the regular image ofa one-session approach. When the digital outputs of the two sub-imagesare combined, the final regular image still preserves the full dynamicrange. Assume that a pixel with full integration time would get ananalog signal to be digitized into 128. By using half the integrationtime for each session, the pixel will get half the analog signal amountand thus will be digitized to 64. The half integration time is importantbecause integration time may be a substantial component in the totalperiod. Therefore, by using only half the integration time for eachsession will cause the total time shorter than otherwise.

FIG. 7A illustrates an example of two-session approach according to anembodiment of the present invention. In the first session, a regularimage can be captured using regular timing. In the second session, amixed image is captured, which includes a structured-light image and atiming-modified regular image. The structured-light image is capturedbetween the two session readouts. The first session readout may betemporally stored in a memory or buffer inside the image sensor oroutside. When the second session readout is complete, the two readoutvalues from two sessions are combined together. Since thestructured-light image is captured between the two sub-images of aregular-scene image, the structured-light image should be very close tothe captured regular-scene images temporally. Accordingly, thestructured-light image should closely correlate to the regular sceneimage. The readouts from two sub-images associated with the two sessionsare combined into one regular image. In one embodiment, a group ofinterleaved sub-regular images and structured-light images are takenconsecutively. For example, the images in odd numbers are for regularimages and images in even numbers are for structured-light images.Images 1 and 3, which are two sub-images, could be combined to form acombined regular image corresponding to image 2, which is astructured-light image. Similarly, images 3 and 5, which are twosub-images, could be combined to form a combined regular imagecorresponding to image 4, which is a structured-light image. Images 5and 7 could be combined to form a combined regular image correspondingto image 6, which is a structured-light image. The process can continuewith each regular sub-image used twice in general. In this case, theweighting factors for images 1, 3, 5, 7, . . . could be 50/50/50/50 . .. , or it could be 60/40/60/40/60 . . . with the principle of the twocombined weighting is 100 percent of the targeted integration time.

In FIG. 7A, the integration time for the two sub-images is roughly thesame. However, the two integration time may also be different. Forexample, the integration time for the first sub-image may be three timesas long as the integration time (also called integration period in thisdisclosure) for the second sub-image as shown in FIG. 7B. In this case,when the digital outputs from the two sub-images are combined, thecombined image has the effect of weighted sum of the first sub-image(i.e., ¾) and the second sub-image (¼). There is no need to perform theweighted sum associated with different integration periods since theweighting will be reflected in the charges accumulated during respectiveintegration periods. The longer integration period results in moreaccumulated charges, which result in a higher analog signal.Accordingly, the sum of the two digital readouts represents the weightedsum of the two sub-images, where the weighting factors correspond to theintegration periods.

In another application of structured-light images, multiple structuredlight images are used to derive more 3D points than a singlestructured-light image for one or more associated regular images. Forexample, multiple structured-light images may be captured consecutivelyby a capsule camera while traversing in the human gastrointestinal (GI)tract. The regular image can be captured between, before or after themultiple structured-light images. The captured structured-light imagescan be used to derive a 3D model of the GI tract. This 3D GI tract modelcan be useful for examining associated regular images of the GI tract.

For two-session regular image capturing with interveningstructured-light image, the means for reducing the frame period for thestructured-light image as mentioned before can be used. For example, thestructured-light image can be captured with a reduced dynamic range ofthe image sensor compared to the first regular image and the secondregular image. The structured-light image may also be captured at lowerspatial resolution than the first regular image and the second regularimage. Furthermore, the structured-light image can be captured with areduced image area in a vertical direction, horizontal direction or bothcompared to the first regular image and the second regular image.

In some cases, the depth or shape information is of interest only for aselected image area. In these cases, the structured-light image can becaptured for the selected image area only. Accordingly, it serves analternative means to reduce the frame period of the structured-lightimage. The reduced image area may correspond to a reduced image area inthe vertical direction, horizontal direction or both compared to theregular image. The means may also be combined with other means, such asreducing the dynamic range or reducing the spatial resolution, forreducing the frame period of the structured-light image.

Reducing the spatial resolution by itself can be used as a technique toreduce the frame period for the structured-light images. For example,the structured-light image can be captured with reduced verticalresolution by only retaining selected rows of pixels and skippingremaining rows of pixels of the image sensor.

For an endoscope application, including a capsule endoscope application,there is no ambient light and the lighting from the endoscope is theonly light source. Therefore, the integration time of each row needs notto be the same as long as the duration of the light exposure is the samefor every line. For the endoscope environment, the lower dynamic rangeof structured-light image than that of the regular image also benefitsfrom temporal proximity between the structured-light image and theregular image. Therefore, the structured-light image according to thepresent invention should bear more accurate depth or shape informationcorrelated with the regular image.

For power sensitive applications such as the capsule endoscope andwearable device, less dynamic range also saves power due to lesscomparison operations and shorter integration time, which requires lessstructured light energy. On the other hand, since signal to noise ratiois not so important to structured-light image, its gain can be set tosubstantial higher to further save energy.

A camera system usually includes an exposure control function to controlthe operating parameters of the image sensor so that the overallintensity of the image taken is at the right level within certain rangeconducive for viewing. The image intensity is derived from the pixelintensity. The detailed control often is subject to the preference ofcamera system designer. For example, the image intensity is determinedby the average of pixel intensity of central portions of the image. Inanother example, the mean of the pixel intensity of the central portionis used as the image intensity. In another example, multiple areas ofthe image are used instead of the central portion. If the intensity isfound to be too high, then the gain or the integration time can bereduced. If the intensity is too low then the gain or the integrationtime can be increased. Furthermore, the amount of adjustment from oneimage to the next can be dependent on how much the intensity is deviatedfrom the preferred level or range.

A camera system may also provide the lighting to augment the ambientlight. The lighting from the camera system may also be the sole lightingsource, such as a regular endoscope or a capsule endoscope. For a cameraused for pipe examination or for deep sea exploration, the lighting fromthe camera is also the sole lighting source. In such a system, theexposure control will control the gain, integration time, lightingintensity and/or energy or a combination of them. If an image has toostrong intensity, the value of (gain×integration×light energy) will bereduced for the subsequent image or images. On the other hand, if animage has too weak intensity, the value of (gain×integration×lightenergy) will be increased for the subsequent image or images. The amountof adjustment from one image to the next may dependent on much theintensity is deviated from the preferred level or range.

An embodiment of the present invention addresses dual exposure controlsfor capturing structured-light images and regular images using a singleimage sensor. Based on this embodiment, there are two exposure controlloops for the same image sensor, one for the structured-light image andthe other for the regular image. In the case that the regular imagelighting is substantially dependent on the light controlled by thecamera system (e.g. negligible or no ambient light), the exposurecondition is very similar for both structured light and the regularlight since the distance to the scene is practically the same for bothcases. Accordingly, one exposure control loop could be used and theother exposure control is dependent on the first exposure control loop.For example, (gain×integration×light energy) of structure light can belinearly dependent on (gain×integration×light energy) of regular lightimage or vice versa. In another embodiment, other dependence is used.For example, gamma-type dependence or dependence on the intensitydistribution may also be used.

In the case where there is ambient light, the structured-light needs tobe sufficiently strong to cause the structure light pattern morediscernable in the structured-light image for analysis. In this case,the light intensity in the above analysis is composed of ambient lightand light or lights projected to the scene controlled by the exposurecontrol of the camera system. In this case, there might be no need forcamera control to project light for regular image if ambient light issufficient. However the structured light has another constraint that theprojected structured-light must be strong enough to show its patternand/or color in the structured-light image. If the spectrum of thestructured light is substantially concentrated in the spectrum of oneparticular color of the image sensor, the intensity of that particularcolor of the structured light image and/or the overall intensity areconsidered. In one embodiment, if structured-light sources are capableof generating multiple colors, then the intensity of each colorcomponent in the regular image is considered. The structured lightsource color corresponding to the weaker color in the regular image ischosen in order to make the structured color stand out or to have ahigher signal to background ratio statistically for easy analysis

FIG. 8 illustrates an exemplary flowchart for capturing a mixedstructured-light image and regular image according to an embodiment ofthe present invention. Capturing a first structured-light image usingthe image sensor during a first frame period in step 810. A regularimage is captured using the image sensor during a second frame period,where the first frame period is shorter than the second frame period andthe first structured-light image is captured before or after the regularimage in step 820.

FIG. 9 illustrates an exemplary flowchart for capturing a tandemstructured-light image and regular image according to one embodiment ofthe present invention, where the structured-light image has lowerdynamic range than the regular image. First reset signals are applied toa pixel array to reset rows of pixels of the pixel array in step 910.The rows of pixels of the image sensor are exposed to structured lightto cause first analog signals from the rows of pixels in step 920. Thefirst analog signals from the rows of pixels of the image sensor areconverted into first digital outputs for the first structured-lightimage using one or more analog-to-digital converters in step 930. Secondreset signals are applied to the pixel array to reset the rows of pixelsof the pixel array in step 940. The rows of pixels are exposed tonon-structured light to cause second analog signals from the rows ofpixels in step 950. The second analog signals from the rows of pixelsare converted into second digital outputs for the regular image usingsaid one or more analog-to-digital converters in step 960. The firststructured-light image is captured before or after the regular image instep 960, where the first dynamic range is smaller than the seconddynamic range.

The invention may be embodied in other specific forms without departingfrom its spirit or essential characteristics. The described examples areto be considered in all respects only as illustrative and notrestrictive. Therefore, the scope of the invention is indicated by theappended claims rather than by the foregoing description. All changeswhich come within the meaning and range of equivalency of the claims areto be embraced within their scope.

1. An integrated image sensor, comprising: a pixel array beingresponsive to light energy received by the pixel array to produce pixelsignals having a voltage level depending on the light energy received bythe pixel array; one or more output circuits coupled to the pixel arrayto access the pixel signals produced by the pixel array; one or moreanalog-to-digital converters having a first dynamic range and a seconddynamic range; one or more timing and control circuits coupled to thepixel array, said one or more output circuits, said one or moreanalog-to-digital converters or a combination thereof, wherein said oneor more timing and control circuits are arranged to: capture, by thepixel array, a structured-light image formed on a common image planeduring a first frame period by applying first reset signals to the pixelarray to reset rows of pixels of the pixel array, exposing the rows ofpixels of the pixel array to first illumination from a structured lightsource to cause first analog signals from the rows of pixels andconverting the first analog signals from the rows of pixels of the pixelarray into first digital outputs for the structured-light image usingone or more analog-to-digital converters; capture, by the pixel array, afirst regular image formed on the common image plane during a secondframe period by applying second reset signals to the pixel array toreset the rows of pixels of the pixel array, exposing the rows of pixelsto second illumination from a non-structured light source to causesecond analog signals from the rows of pixels, and converting the secondanalog signals from the rows of pixels into second digital outputs forthe first regular image using said one or more analog-to-digitalconverters; capture, by the pixel array, a second regular image formedon the common image plane during a third frame period by applying thirdreset signals to the pixel array to reset the rows of pixels of thepixel array, exposing the rows of pixels to the second illumination fromthe non-structured light source to cause third analog signals from therows of pixels, and converting the third analog signals from the rows ofpixels into third digital outputs for the second regular image usingsaid one or more analog-to-digital converters; combine the seconddigital outputs and the third digital outputs to form a combined regularimage; and wherein the structured-light image is captured between thefirst regular image and the second regular image to derive depth orshape information for the combined regular image.
 2. The integratedimage sensor of claim 1, wherein said one or more analog-to-digitalconverters correspond to a configurable analog-to-digital converter toprovide both the first dynamic range and the second dynamic range. 3.The integrated image sensor of claim 1, wherein said one or moreanalog-to-digital converters correspond to two analog-to-digitalconverters with two different resolutions to provide both the firstdynamic range and the second dynamic range.
 4. The integrated imagesensor of claim 1, wherein said one or more timing and control circuitsare further arranged to cause the structured-light image to be capturedat lower spatial resolution than the first regular image and the secondregular image.
 5. A camera system, comprising: a structured lightsource; a non-structured light source; and an integrated image sensor,comprising: a pixel array being responsive to light energy received bythe pixel array to produce pixel signals having a voltage leveldepending on the light energy received by the pixel array; one or moreoutput circuits coupled to the pixel array to access the pixel signalsproduced by the pixel array; one or more analog-to-digital convertershaving a first dynamic range and a second dynamic range; one or moretiming and control circuits coupled to the pixel array, said one or moreoutput circuits, said one or more analog-to-digital converters or acombination thereof, wherein said one or more timing and controlcircuits are arranged to: capture, by the pixel array, astructured-light image formed on a common image plane during a firstframe period by applying first reset signals to the pixel array to resetrows of pixels of the pixel array, exposing the rows of pixels of thepixel array to first illumination from the structured light source tocause first analog signals from the rows of pixels and converting thefirst analog signals from the rows of pixels of the pixel array intofirst digital outputs for the structured-light image using one or moreanalog-to-digital converters; capture, by the pixel array, a firstregular image formed on the common image plane during a second frameperiod by applying second reset signals to the pixel array to reset therows of pixels of the pixel array, exposing the rows of pixels to secondillumination from the non-structured light source to cause second analogsignals from the rows of pixels, and converting the second analogsignals from the rows of pixels into second digital outputs for thefirst regular image using said one or more analog-to-digital converters;capture, by the pixel array, a second regular image formed on the commonimage plane during a third frame period by applying third reset signalsto the pixel array to reset the rows of pixels of the pixel array,exposing the rows of pixels to the second illumination from thenon-structured light source to cause third analog signals from the rowsof pixels, and converting the third analog signals from the rows ofpixels into third digital outputs for the second regular image usingsaid one or more analog-to-digital converters; combine the seconddigital outputs and the third digital outputs to form a combined regularimage; and wherein the structured-light image is captured between thefirst regular image and the second regular image to derive depth orshape information for the combined regular image.
 6. The camera systemof claim 5, wherein the first frame period is shorter than a sum of thesecond frame period and the third frame period.
 7. The camera system ofclaim 5, wherein the structured-light image is captured at lower spatialresolution than the first regular image and the second regular image. 8.The camera system of claim 5, wherein said one or more analog-to-digitalconverters correspond to a configurable analog-to-digital converter toprovide both the first dynamic range and the second dynamic range. 9.The camera system of claim 5, wherein said one or more analog-to-digitalconverters correspond to two analog-to-digital converters with twodifferent resolutions to provide both the first dynamic range and thesecond dynamic range.
 10. An endoscope for in vivo viewing of humangastrointestinal (GI) tract, comprising: a structured light source; anon-structured light source; an integrated image sensor, comprising: apixel array being responsive to light energy received by the pixel arrayto produce pixel signals having a voltage level depending on the lightenergy received by the pixel array; one or more output circuits coupledto the pixel array to access the pixel signals produced by the pixelarray; one or more analog-to-digital converters having a first dynamicrange and a second dynamic range; one or more timing and controlcircuits coupled to the pixel array, said one or more output circuits,said one or more analog-to-digital converters or a combination thereof,wherein said one or more timing and control circuits are arranged to:capture, by the pixel array, a structured-light image formed on a commonimage plane during a first frame period by applying first reset signalsto the pixel array to reset rows of pixels of the pixel array, exposingthe rows of pixels of the pixel array to first illumination from thestructured light source to cause first analog signals from the rows ofpixels and converting the first analog signals from the rows of pixelsof the pixel array into first digital outputs for the structured-lightimage using one or more analog-to-digital converters; capture, by thepixel array, a first regular image formed on the common image planeduring a second frame period by applying second reset signals to thepixel array to reset the rows of pixels of the pixel array, exposing therows of pixels to second illumination from the non-structured lightsource to cause second analog signals from the rows of pixels, andconverting the second analog signals from the rows of pixels into seconddigital outputs for the first regular image using said one or moreanalog-to-digital converters; capture, by the pixel array, a secondregular image formed on the common image plane during a third frameperiod by applying third reset signals to the pixel array to reset therows of pixels of the pixel array, exposing the rows of pixels to thesecond illumination from the non-structured light source to cause thirdanalog signals from the rows of pixels, and converting the third analogsignals from the rows of pixels into third digital outputs for thesecond regular image using said one or more analog-to-digitalconverters; combine the second digital outputs and the third digitaloutputs to form a combined regular image; and wherein thestructured-light image is captured between the first regular image andthe second regular image to derive depth or shape information for thecombined regular image.
 11. The endoscope of claim 10, wherein thestructured-light image is captured at lower spatial resolution than thefirst regular image and the second regular image.
 12. The endoscope ofclaim 10, wherein said one or more analog-to-digital converterscorrespond to a configurable analog-to-digital converter to provide boththe first dynamic range and the second dynamic range.
 13. The endoscopeof claim 10, wherein said one or more analog-to-digital converterscorrespond to two analog-to-digital converters with two differentresolution to provide both the first dynamic range and the seconddynamic range.
 14. The endoscope of claim 10 further comprising acapsule housing adapted to be swallowed, wherein the pixel array, thestructured light source, the non-structured light source, said one ormore output circuits, and said one or more timing and control circuitsare enclosed and sealed in the capsule housing.